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ABSTRACT 


A group membership protocol provides the mechanisms to ensure the consistent 
group views among a group-oriented distributed processes. The protocol is required to 
dynamically re-configure the group views among the various members in the event of a 
change to the group due to a new member joining or a member departing. The departure 
may be voluntary or involuntary. The protocol must provide a scheme to detect the failure 
of any of the members and re-configure the group. Multiple changes to the group must be 
perceived at all members in the same order. 

This thesis deals with a particular group membership protocol. The protocol 
structures the group as a logical ring. Changes to the group are accomplished using a 
two-phase scheme. The agreement phase consists of circulation of an agree token. 
Processing the token makes a pending change known to all members. The commit phase 
incorporates the changes in the correct order. 

This thesis presents an implementation of this asynchronous group membership 
protocol. The main feature is that the decentralized nature of the protocol eliminates the 
need for a dedicated coordinator of changes. The processing requirements for the 
protocol are likewise distributed. The processing time required to implement a change to 


the group is shown to have a linear relationship to the group size. 
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I. INTRODUCTION 


A. BACKGROUND 
1. Distributed Computing 


Distributed computing is at the forefront of today's computing research. The 
increased reliability and performance has resulted in distributed computing being used i 
various applications such as distributed control applications, distributed databases, and 
real-time settings [1]. The need for fault-tolerant systems is particularly important to 
military applications. Such applications typically require fault tolerant algorithms, 
real-time response, on-line reconfiguration, and other schemes to increase reliability. 
These requirements, however, lead to significant additional complexity. This complexity 
arises in part due to the reliable inter-processor communication required to implement the 
distributed processing. Networxs are inherently unreliable and make reliable application 
level message passing a non-trivial task. One of the primary requirements of reliable 
distributed applications is a reliable multicast communication primitive. A group 
membership protocol simplifies the construction of a such a primitive [2]. 

2. Group Membership 

Cooperating processes constituting a single application share resources and 
constitute a process group. Underlying the consistent behavior of such a group is the 
requirement that all members of the group have implicit knowledge of all other members 
in the group. Additionally, all members must perceive changes to the group in the same 
order. Group membership will change as processes join >r depart the group. Departures 
may be voluntary or involuntary as in the case of a failure. An additional requirement is 
the timely detection of members failing. In order to enhance the robustness of the system, 


intra-member monitoring must occur. This can be a simple exchange of messages 


indicating that the process being monitored is still "live." 





Historically, protocols solving the group membership problem have been of a 
centralized design. One member acted as the group host and had the responsibility of 
monitoring all subordinate members. All changes were detected by the host. Upon a 
change to the group view, the host broadcast the new group view to all members of the 
group. Obviously, problems can arise if the host itself fails. Voting must occur in order to 
elect a new host with the added overhead associated with the voting. Additionally, the 
processing requirements are unequally distributed between the members of the group as 


the host takes a major share of the load. 


B. SCOPE AND CONTRIBUTION 


In this thesis, an implementation of the decentralized asynchronous membership 
protocol is presented. The protocol was originally presented in (3] and [4]. It was further 
refined and partially implemented in [5]. This thesis presents a brief overview to the 
protocol. A more detailed explanation of the protocol and definitions can be found in [5]. 
This thesis covers additional refinements to the protocol necessary to correct flaws 
discovered in the coding and testing phase of development. Additionally, the performance 
of the protocol on an Ethernet Local Area Network (LAN) is presented and discussed. 


C. ORGANIZATION OF THE THESIS 


The thesis is divided into six chapters. Chapter II presents an overview of the basic 
operation of the protocol. Chapter III discusses the changes required to implement 
improvements to the protocol. Chapter IV covers possible further refinements to the 
protocol. In Chapter V, the performance of the current protocol is analyzed. Chapter VI 
presents the analysis and possible future areas of research. The code developed is 


included in the Appendix. 


II. GROUP MEMBERSHIP PROTOCOL 


In this chapter, the group membership protocol (GMP) is described. The original 
protocol was presented in [3] and further developed in [5]. This chapter presents an 


overview of the protocol. 


A. GROUP MEMBERSHIP PROTOCOL OVERVIEW 
1. Assumptions 


The following assumptions are made by the GMP in order for proper 
implementation. A fully-connected network of reliable First-In / First-Out (FIFO) 
communication channels connecting operational members is assumed. All failures are 
assumed to be crash or fail-stop[6]. This implies that a message sent will be delivered 
unless the recipient has failed. However, there is no upper bound on the time ox delivery. 

Multiple changes to the membership are allowed simultaneously. However, the 
changes are committed one at a time and in the same order at all members. 

A member is added to the group when a join is processed and is removed when 
a failure is perceived. 

The group name is assumed to be public. Those elements that may wish to 
become members by joining the group have access to the group file which contains the 
current group view. A prospective member searches for the file on a given site. The 
group view is extracted and the prospective member sends the joinregst to the appropriate 
address. 

The protocol maintains three main databases at each member; the membership 
list (group view), status table and token pool. Each has a separate database manager to 
ensure mutual exclusion to all processes needing the services of the database. 

2. Overview 
The GMP guarantees that the group view changes occur in the same relative 


sequence at all operational members of the group. 





The most significant feature of the GMP is the decentralization. No single 
member is responsible for detecting a change to the membership nor guaranteeing group 
view consistency among the group members. A logical ring is used to implement both of 
these functions in a distributed manner. The logical ring is a circular ordering of the 
members of the group. 

The physical location of the member has no relation to the ordering of the 
logical ring. Within the ring structure the direction of traversal was arbitrarily chosen as 
clockwise. Given the structure each member only monitors its anti-clockwise neighbor, 
the acwnbr. The acwnbr responds to a status query, statusgry, with a status report, 
statusrpt. Likewise, it sends a statusgry to its acwnbr. Thus, every member monitors 
only one other member of the group and is itself monitored by it clockwise neighbor, 
cwnbr. 

Consider a five member group. Process p, was the initial member of the group. 
The other members joined in an order such that p, is the acwnbr of p,, p, is the acwnbr of 
P,, and so on. Member p, sends a statusgry to py, which responds with a statusrpt. 
Likewise, p, sends a statusgry to p,. This is illustrated in Figure 1. For clarity, only the 
monitoring and response is shown for the first set of neighbors. 

The ring configuration changes as the group membership changes. The ring 
starts as a single member group with other members joining in some arbitrary order. All 
changes to the group are treated in a similar manner. Members wishing to join an existing 
group read the group membership file in the first active member located on the net. The 
joining member then sends a joinregst to the first member of the group. If the initial 
parameters message, initparams, is not received in a reasonable time, the request is 
transmitted to the next member in the group view file until all members have been 
attempted or a successful join has been completed. 

A failure is considered an involuntary departure. When a member departs the 
group voluntarily, it simply stops responding to statusgrys. It will then be perceived as 
failed and subsequently removed from the group. Delayed transmission of the statusrpt, 








failure of the member to respond to the statusgry and a lost statusrpt will all result in the 
monitored member being detected as failed. 


direction of ring transversal 





Figure 1 A Logical Ring 


3. Processing of Individual Changes 

The GMP allows for a two phase procedure for all changes to the group view, 
the agree phase and the commit phase. An agree token is circulated around the ring. 
Once the originator receives the token via the ring, all members have processed the agree 
token. At this point the agree token is converted to a commit token, and the change 
agreed upon is committed by each member as the token is circulated around the ring. The 
protocol ensures that each token is received by all members, processed only once, and 
never lost. More complete descriptions of the actions required by the different phases is 


covered in the following section. 





a. Departure Processing 

Once a member perceives the departure of its acewnbr, voluntary or 
otherwise, a failagree token is initiated. The failed member is added to the status table 
with a failagree status. The token is incorporated into the token pool and transmitted via 
the FIFO channel to the cwnbr. Similar processing occurs at members receiving a 
failagree token for the first time. Once the token has been received by the originator, the 
agree phase has been completed. The failed member is removed from the status table and 
group view. The token pool is purged. The failcomit token added to the token pool and 
transmitted around the ring. When a member receives a failcomit token for the first time, 


similar processing occurs. 
b. Join Processing 


The protocol maintains a logical marker between the first and last members 
to join the group. The first member is called the host. A new member will always join the 
group as the acwnbr of the host. The host has the responsibility of initiating the joinagree 
token for the new member. However, the host may not be the member of the group that 
receives a join request message from the new member. In this case, the message is 
converted to a token, forwarded, and the new member is added to the status table. Once 
the host has received either a joinregst token or message, it initiates a joinagree token. 
The host then adds the token to the token pool, the member to the status table, and 
transmits the token to its cwnbr. Similar processing occurs when a member receives a 
joinagree token for the first time. When the host receives the joinagree token via the 
token ring, it initiates the commit phase. 

The joincomit phase consists of purging the token pool, incorporating the 
joincomit token in the token pool, adding the new member to the group view, and 
forwarding the token. Additionally, the host will transmit the status table, group view, 
and token pool to the new member in the initial parameters message, initparams. This is 


accomplished by IntegrateMember. 








Il. PROTOCOL CHANGES 


This chapter describes the revisions to the group membership protocol proposed in 
[5]. Modifications discussed include lost messages, delayed transmission, lost token 


acknowledgments, and proper termination of the agree phase. 


A. LOST INITIAL PARAMETERS MESSAGE 
1. Problem 


Consider a lost initial parameters initparam message. After the initparam 
message is sent, the joining member is regarded as part of the group by the sender even if 
it is lost. However, the group membership, token pool and status table are not accurately 
reflected in the joining member's local database. There was no mechanism for 
re-transmission of the initparam message, nor was it possible to recreate the information 
locally. 

2. Solution 

The status reporter is required authenticate the group membership. Reports are 

generated in response to status queries from only those members that are in the group or 


are joining the group. Status queries from outside the group are ignored. See Figure 2. 


ReportStatus process at p; 


if (not blocked by IntegrateMember) 
if (querying member € GV,, or has joinagree status) 


1 
2 
3 Pron = querying member 

4 send status to p,_,, 

5 if (previous querying member = p,,,,) 
6 send TokenPool(p,) to p,,, 

7 
8 
9 


end ReportStatus 





Figure 2 Reporting of Status 








3. Justification 
A lost initparam message will result in the new member failing to respond to 
the host's first statusgry. Upon time out, the new member will be considered a failed 
process. The host's original aewnbdr will then be monitored anew by the host. The new 
member, never having received the initparam message will time out on the join request. 
It will then attempt to join again. 

By the time the initparam message is transmitted, the host's original acwnbr 
has knowledge that the new member is now joining the group. Therefore, with this 
change, the host's original acwnbr issues Status reports to the new member's queries. 

4. Side effects 

Such group authentication will prevent multiple switching of the cwnbr. If the 
initparam message is lost, the host's original acwnbr will be un-monitored for a short 
time. Upr the failure detection of the new member, the host's original acwnbr will again 
be monitored by the host process. To ensure that the StatusReporter responds only to 
members within the current group view, IntegrateMember must be atomic with respect to 


StatusReporter. This will prevent race conditions when initparam message is received, 


followed by an almost simultaneous receipt of the first status query from the host process. 


Figure 3 shows the process dependencies, while Figure 4 depicts the specification for 
IntegrateMember. The inter-process dependencies for the monitor processes are shown 


in Figure 5. 





Commit 
Processor 
Group View | View_Request : Status Table 
Manage es Ue 
Status_ Table 


INTEGRATE 
MEMBER 


Figure 3 Integrate Member - Process Dependencies 


IntegrateMember 


if (initial parameters) 
send blocking message to status reporter 
send GV,, to group view manager 
send unblocking message to status reporter 
send ST), to status table manager 
send TokenPool(p,) to token pool manager 
else 
get GV,, from group view manager 
get ST,, from status table manger 
get TokenPool(p,) from token pool manager 
assemble initparam message 
send initparam message to new member 
13 end 


Figure 4 Integrate Member Process Specification 
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Figure 5 Monitor Process - Internal Structure and Dependencies 


B. DUPLICATE PROCESSING 
1. Problem 

The protocol was designed for processing in a member to be concurrent. 
Consider AgreeProcessor and ComitProcessor. It is possible for the AgreeProcessor to 
receive an agree token immediately followed by an external token pool containing the 
same token. The token may be converted to a commit token and forwarded to 
ComitProcessor. Due to context switching, processing of the commit token may not be 
immediate. Further, AgreeProcessor begins processing the token pool which contains the 
copy of the original agree token. Since the processing of the commit token has not 
occurred, it is possible for the agree token from the token pool to be detected as requiring 


10 








conversion. The subsequent processing of the duplicate commit is an error. Figure 6 
depicts the problem. 
Time AgreeProcessor 


receive failagree token 


process failagree 


send init failcomit 
= receive init failcomit 


receive external token pool 
process failagree process failcomit 
- update status 
- purge token pool 





error condition - commit change 





Figure 6 Error Condition Arising from Asynchronous Programs 


2. Solution 
In order to solve this problem, CommitProcessor must be atomic with respect 
to AgreeProcessor. (see Figures 7 and 8) 
3. Justification 
Both processors use the same databases. Rejection of duplicate tokens depends 
upon the current state of the databases. Since token rejection is accomplished by 
AgreeProcessor, and ComitProcessor updates the state to reflect the commit in progress, 
AgreeProcessor must not begin its next iteration until ComitProcessor has updated the 
state. ComitProcessor does not fully update the state until just prior to transmitting the 


commit token around the ring. Refer to Figure 9, lines 1-3. 
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Figure 7 Agreement Processor - Process Dependencies 
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CommitChange for commit, {p,) at p; 


/* Depending on whether a join or departure */ 

1 add or delete p, from GV,, 

2 delete p, entry from ST, 

3 vn(p,) <— va(p) + 1 

4 delete all commit tokens received before agree,(p,) from TokenPool(p,) 
5 if Goin committed && joinreg,(p,) € TokenPool(p,) ) 
6 delete joinreq,(P,) 

7 end 

8 add commit, {Pp,) to TokenPool(p) 

9 delete agree,(p,) 

10 if (current host = p,) 

11 determine new p,,, 

12 end 

13 if (Goin committed) && (p,,, = P,)) 

14 send ST,,, TokenPool(p,), and GV,, to acwnbr(p,) 
15 end 

16 send commit,{p,) token to cwnbr(p,) 





end CommitChange 


Figure 9 Actions for Committing a Change 


C. INVALID DELETE TOKEN 
1. Problem 
An attempt to delete a nonexistent joinregst token from the token pool would 
result in an error condition and would hang the process. The joinregst token is not always 
present in the token pool. The token occurs only if a joining member has made the 
request to a member that is not the host. 
2. Solution 
Before attempting to delete the joinregst token, the token pool is checked. If 
the token is present, it is then deleted. Figure 9, lines 5-7. 














3. Justification 
This is a special case token, and does not always occur in all token pools. 
Exception handling as above will correct any incor.sistencies among the various token 


pools. 


D. LOST TOKEN ACKNOWLEDGMENT 
1. Problem 
If the token acknowledgment is not received by the sending front process, the 
token will remain on the queue. It will be re-transmitted every time that front receives a 
message. The original token will be processed at the receiving end and further receipts of 
the «ame token will be detected as duplicates and discarded. The problem lies in that the 
queue is ..ever cleared. Therefore, subsequent tokens will be blocked behind the token for 
which the acknowledgment was lost, unless a failure of one of the two members occurs. 
2. Solution 
This problem is solved by checking the serial number of the message on the 
receiving end of the FIFO channel. If the message is the last token received, the token 
acknowledgment is re-transmitted . 
3. Justification 
If the message received is the expected token, an acknowledgment is sent. The 
token is forwarded to the appropriate internal sub-process. If the last token received is 


received again, a token acknowledgment is sent back and the duplicete token is discarded. 


This mechanism will account and correct for lost token acknowledgments. Figure 10, 


lines 17-19. 








FIFO Channel - BACK process 





1 Wait for a channel ready to ready 
2 if (internal channel ready) 

3s if (Status Query) 

4 update acwnbr 

5 send Status_Query 

6 else if (Jnitial_ Parameters) 

7 update acwnbr 

8 send /nitial_ Parameters 

9 else if (Join_Request) 

10 send Join_Request 

11 end 

12 else /* external channel ready */ 
13 if (message originator = acwnbr) 
14 if (Status Report) 








15 send Status_Report to MONITOR_PROCESS 

16 else if (Token) 

17 if (Serial_ Number = Expected_Serial_Number - 1) 
18 send Token_Ack /* to acwnbr */ 

19 end 

20 if (Serial_ Number = Expected_Serial_Number ) 
21 send Token to AgreeProcessor 

22 send Token_Ack /* to acwnbr */ 

23 increment Expected_Serial_Number 

24 end /* out of order messages are discarded */ 

25 else if (Token_Pool) /* Token_Pool is always accepted */ 
26 send Token_Pool to AgreeProcessor 

Zz send Token_Ack /* to acwnbr */ 

28 set Expected_Serial_ Number = Serial_ Number + 1 
29 end 

30 «end 


Figure 10 FIFO Channel - Back Process 


E. AGREEPROCESSOR 


The specification for the AgreeProcessor was rewritten to account for various 
subtleties and to improve overall readability. All tokens received through the FIFO 


channel layer are sent to AgreeProcessor for dispatching to the appropriate processor. A 
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duplicate token is one that has been previously processed at given member. Some scheme 
to detect and reject duplicate tokens is required. Proper termination of the agree phase 
and subsequent initiation of the commit phase are also required. Figure 11. In this 


section, we discuss the operation of the agree processor. 





























AgreeProcessor for agree,(p,) at p; 


1 if (not blocked by CommitProcessor) 

2 _ if (initiate agreement message received) /* p; = p, */ 
3 add agree,{p,) to TokenPool(p;) 

4 ST,({p,) <— joinagreed or failagreed 

5 send agree,(p,) to cwnbr(p,) 

6 send acknowledgment to calling process 

7 else /* a token or external token pool is received */ 
8 if (ExtTokenPool) 

9 for Vtokens € ExtTokenPool 

10 if (token € TokenPool(p,)) 

11 if (originator failed) 

12 ProcessToken 
13 end 
14 else /* token not in TokenPool */ 
15 if (received for the first time) 
16 ProcessToken 
17 end 
18 end 

19 end 

20 else /* a token was received */ 
21 if (received for the first time) i 
22 ProcessToken 
23 end | 
end 





Figure 11 Agreement Processor 


1. Initiate_Agreement Message 
When AgreeProcessor receives an initiate_agreement message, the appropriate 


agree token is generated, added to the token pool and forwarded to the cwnbr. The 
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status table entry for the subject is updated. An acknowledgment is returned to the calling 
process. This reflects no change to the prior specification. 
2. External Token Pool 

When an external token pool is received, it is compared to the local token pool. 
All tokens in the external token pool are examined. Processing of a token depends on 
whether the token is also present in the local token pool. 

If an agree token from the external token pool is in the local token pool, the 
token originator may have failed. Due to a failure of the originator, the agree token is 
requires Conversion into a commit token at the first active clockwise neighbor of the 
originator only. Such tokens are processed as if they had been received as a separate 
token message. If the token is not present, it may have already been purged. These 


tokens must be rejected as duplicates. 
a. Token Criginator Failed 


Detection of the token originator failing prior to initiating the commit 
phase is accomplished separately for joinagree and failagree tokens. Figure 12. It is 
necessary for the next active member to detect the originator's failure and also initiate the 
commit phase for the incomplete change started by the originator. The agree token may 
be received as part of the external token pool. The token will also be present in the local 
token pool from the acwnbr of the failed originator. This situation may also occur if a 
member in the middle of the ring fails. The failed member's cwnbr will receive an external 
token pool containing the original agree. However, this does not require a commit to be 
initiated as the originator has not failed. It is essential that the differences be noted and 
accounted for. The same conditions are present, but different processing must occur. 

The originator's failure during a joinagree phase can be detected if the rank 
of the external token pool originator is greater than the rank of the current member. 
Consider the host failing prior to initiating the commit phase. All members in the group 
have agreed to the join. The joinagree token will be received by the new host upon ring 


reconfiguration. The new host's rank is zero (Q) while the external token pool originator's 
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rank is the (group size - 1). Since Rank(originator) > Rank(host), a commit must be 
initiated. 

Consider member p, with rank i failing during the joinagree phase. When 
the failure of p, is detected, p,,, receives the external token pool from p,,. The joinagree 
token is present in both the external token pool and TokenPool(p,,,). The token is 


rejected since Rank(p,,) < Rank(p,,,), . 


LostAgreeToken 
if (joinagree) 
if (rank(p,) > rank(p,)) 
return true 
else 
return false 
end 
end 


if (failagree) 
if (RelutiveRank(p, , p,) > RelativeRank(p, , p,)) 
return true 
else 


return false 





Figure 12 Determination of Token Originator's Failure 


Now we consider a duplicate failagree token. Define RelativeRank(p, P,) 
as the rank of p; with respect to p; instead of the host. A ring transversal starts and ends at 
the same specified process, i.e. any given member follows itself in a ring transversal. 
Figure 13. RelativeRank for a process that is not a member of the group is undefined. 
Recall the subject of a failagree remains a member of the group view until the commit is 
processed. A lost failagree token is determined by the RelativeRank of the token subject 
p, and token pool originator p,,.. 
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Process Rank 


RelativeRank (with respect to process #2) 





Figure 13 Relative Rank 


If the RelativeRank(p,, p,) 2 RelativeRank(p,,,, p,) the token originator has 
failed and the failcomit phase should be initiated. This situation will occur if the failed 
token originator itself fails prior to initiating the commit phase. 

b. Duplicate Processing 

Conditions to detect if a token has been received and processed already are 
summarized in Table 1. It is necessary to detect duplicate processing if the external token 
pool contains tokens not found in the local token pool. There are two possible ways for 
this to occur. If the token has not been received and processed, or the token has been 


purged from the local token pool. 











Table 1 CONDITIONS TO DETECT DUPLICATE PROCESSING 


joinregst 


joinagree 


joincomit 


Sailagree 


failcomit 





Consider the commit phase. All members of the group have agreed to a 
particular ring reconfiguration. Due to the latency of token transmission around the ring, 
not all members commit the change simultaneously. Recall the token pool is purged of old 
commit tokens and the corresponding agree token prior to the commit token 
transmission. If a member that has committed a change receives an external token pool 
from a member that has agreed to the change, the agree token remains in the external 
token pool. Likewise, a previous commit token may be received as part of the external 
token pool. Since the tokens have been processed and removed from the local token pool, 
it is necessary to check the effects these tokens may have had on the group view, had they 
been previously processed. For duplicate join tokens, the subject would already be a part 
of the group view. Conversely, the subject of duplicate fail tokens would have already 
been removed from the group view. In this manner, duplicate processing can be detected 
and the tokens rejected. 

It is not necessary to include all possible conditions for a token having been 
processed. Recall the duplicate processing check occurs only if the token is not present in 
the local token pool. For a joinregst, if the token is received as part of the external token 
pool and has been purged from the local token pool, the joincomit must have occurred. 
Since the result of the joincomit is subject becoming a part of the group, it is only 
necessary to check the end result. Intermediate stages of the join process have not 


deleted the joinregst token from the local token pool. Since the token remains in the 
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token pool, it is in both local and external token pools and is rejected. Similar logic results 
in the conditions presented in Table 1. 
3. Tokens 
Each type of token is handled individually upon receipt by AgreeProcessor. 
Figure 14. AgreeProcessor acts as a filter to remove duplicate tokens before forwarding 


non-agree tokens to the appropriate processor. Agree tokens are processed locally. 
a. Joinregst Tokens 


The joinregst tokens are forwarded to JoinProcessor for further 
processing. 
b. Agree Tokens 
If the current process is not the originator of the token, and it is not part of 
the local token pool, it is added to the token pool, the status updated and the token sent to 


the cwnbr. This accounts for the first time an agree token is received and processed. 
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ProcessToken 
1 if (oinregst) 

2 send token to JoinProcessor 
3 elseif (commit) 

4 send token to ComitProcessor 
5 elseif (agree) 
6 
7 
8 

















if ((p, # P) && (agree token ¢ TokenPool(p,)) 
add agree,{p,) to TokenPool(p,) 
ST,(p,) <- FailAgreed or JoinAgreed 


9 send agree,(p,) to ewnbr(p,) 

10 else P; 

il if ((p;= p) || (Vp, ! p>; p, € ST),)) 

12 compute rank Vp,é€ ST), with Agreed status 
13 if rank(p,) = smallest 

14 send initiate_comit to ComitProcessor 
15 else 


ST,(p,) <— joinpendg or failpendg 





Figure 14 Processing Agree Tokens 


If the current process is the token originator, or the first active process 





clockwise from the originator that receives the agree token after it circulates around the 
ring, the rank of all processes with an agreed status is computed. An initiate_commit is 
transmitted if the subject is the lowest ranked of all processes fulfilling the above 


conditions. If not, the subject's status is updated to pending. 
c. Commit Tokens 


A commit token is immediately sent to CommitProcessor for processing. 
4. Side Effects 
The external token pool mentioned above requires a new message type, 











ExTknPool. Figure 15. The message includes the originator of the token pool as an 
internal field. This is required by AgreeProcessor for determination of a failed token 


originator. 


EXTERNAL FORMAT 


SERIAL |,,| MESSAGE ff 
NUMBER| \ |ORIGINATOR] “| €*: 


TOKEN FIELD 


sp = Space character TOKEN TOKEN TOKEN 
\n = New line character TYPE SUBJECT|SP} ORIGINATOR 





Figure 15 External Token Pool Message Format 


F. MULTIPLE JOINS 
1. Problem 


Consider two members attempting to join a group almost simultaneously. 
Figures 16-18. The first member's join will be completed properly. Upon completion of 
the first join, it is possible to begin processing the second member's joinregst before the 
FIFO channels reflect the re-configured ring with the first member in it. It is possible to 
complete the second join before the channels are re-configured. This can happen because 
of the de-coupled protocol and FIFO channels. The host does not change its acwnbr until 
it initiates a status query to the last member. The FIFO channel determines a member's 
acwnbr as the target of the most recent statusury. The cwnbr is the originator of the 
most recent statusgry received. There is an inherent latency involved in the FIFO channel 
reconfiguration due to the timing considerations of subsequent statusgrys. Thus, it is 
possible for a joinregst from a second new member to complete both phases of the join 
process prior to FIFO channel reconfiguration. The first new member may never have 
processed the joinagree and joincomit for the second joining member prior to the second 
member being incorporated into the group view. When the first joining member 
determines its acwnbr and receives the external token pool, the joincomit token for the 
second member may be present. Processing the token will result in attempts to delete the 
corresponding nonexistent joinagree token never received by the first member and 
subsequent removal of the second member from the status table. Both of these are error 


conditions. 
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Figure 16 Simultaneous Joins 


2. Solution 
Initiate the FIFO reconfiguration upon transmission of the initial parameters to 
the new member. Figure 10, line 7. 


CaS) 


Figure 17 Group View after 1 member has joined 
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3. Justification 
The FIFO channel can be considered to be re-configured when the host 
determines its acwnbr. The response from the new acwnbr is not required, as the FIFO 
channel reject all tokens unless they are sent by the acwnbr. Tokens from the old acwnbr 
are rejected and must be processed by the new acwnbr prior to being forwarded to the 


host member. 


Figure 18 Group View at Host & J2 


G. OTHER IMPROVEMENTS 


In this section, oversights to the protocol specification and the modifications required 
are briefly described. 


1. Joinregst Token Processing 
a. Problem 
When processing a join message from a prospective member, InitiateJoin 


adds a joinregst token to the token pool prior to generating it. 





b. Solution 


If the current message being processed is a join message, generate the 


joinregst token and then add the token to the token pool. Figure 19, lines 10-13. 


InitiateJoin for a join request message/token for p_, at p, 


while (true) 
if (p_, € STp, GVp,) 
receive join request message or token for p__, 
end 
if (D; = Pros) 
send initiate agreement message to AgreeProcessor for p,_, 
block until AgreeProcessor acknowledges end of processing 
else 
ST (Pam) <— JoinRequested 
if (join request message) /* p__, locates p, and sends its join request */ 
generate joinreq,(P_,) token 
end 
add joinreq,(p,,,) to TokenPool(p,) 
send joinreq token to cwnbr(p,) 
end 


Figure 19 Processing of a Join Request Message / Token 
2. Commit Token Generation 
a. Problem 
If a member was in a pending status, a commit token for that process was 
never generated prior to committing the change. These members would remain pending 
indefinitely. 
b. Solution 
Create the commit token before committing the change for a member with 


a pending status. Figure 20, lines 11-12. 





ProcessCommitTkn for commit,{p,) at p, 





1 if (initiate commit message received) 

2 generate commit token 

3 token to be processed < generated token 

4 elseif ((p,#p) && (not duplicate) 

Bs token to be processed < received token 

6 else 

Z exit 

8 end 

9 CommitChange 

10 while ( p,¢ ST,, with pending status & Rank(p,) < Rank(p,), p, € ST>,) 
11 generate commit token 

12 token to be processed < generated token 

13. CommitChange in rank order 

14 end 






Figure 20 Generate / Receive and Process a Commit Token 


3. Message Queue in the FIFO Channel Layer 
The front processor was modified to transmit the head of the message queue 


after receiving any message, either on the internal or external channel. Figure 21. 
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FIFO Channel - FRONT Process 





1 Wait for a chant.el ready to read 

2 if (external channel ready) 

3 if (Status_Query) 

4 send Status_Query to MonitorProcess 

5 else if Join®equest) 

6 send JoinRequest to JoinProcessor 

7 ~~ else if (/nitialParameters) 

8 send /nitialParameters to JoinProcessor 
9 else if (TokenAck) 





10 if (Received_Serial_Number = Expected_serial_number) 
11 remove Head_of_Queue 

12 decrement Queue_Counter 

13 end 

14 end 


15. else /* internal channel ready */ 
16 __ if (Token) 


17 change Token to external format /* add external header */ 
18 insert Token in queue 

19 increment Serial_Number 

20 increment Queue_Counter 

21 else if (TokenPool) 

22 discard all messages in queue 

23 change TokenPool to external format /* add external header */ 
24 insert TokenPool in queue 

25 increment Serial_Number 

26 increment Queue_Counter 

27 ~__selse if (StatusReport) 

28 update cwnbr 

29 send StatusReport to cwnbr 

30 end 

31 end 


32 if (Queue_Counter > 0) 

33 send Head_of_Queue to cwnbr 

34 set Expected_serial_number = Head_of_Queue_serial_number 
35 end 





Figure 21 FIFO Channel - Front Process 
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H. SYNOPSIS 

This chapter has described the changes that were required to successfully implement 
the group membership protocol. Changes covered coding as well as protocol related 
problems not discovered in the original specification. These changes deal with the correct 
functioning of the protocol and do not address performance issues. Performance is dealt 


with in Chapter IV. 
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IV. PERFORMANCE OF THE GMP 


The performance of the protocol on the Electrical and Computer Engineering Local 
Area Network (ECE LAN) consisting of SUN2 workstations connected via an Ethernet, 


was measured and the results are presented in this chapter. 


A. LATENCY 
The latency involved in processing changes to the group view is measured by each 
member using the local time clock on each specific processor. Timestamps were 


generated for the conditions listed in Table 2. 


Table 2 CONDITIONS WARRANTING A TIME STAMP 
Time Stamp Where Taken in Code 


initiate an agree token 


receive an agree token 


send a commit token 





The timestamps and related data were dumped to a file local to each processor. A 
filter program was designed and written to compile the data from the various systems, 
given a list of the members in the test group, and the maximum number of members. The 
filter program was designed for a restricted set of all possible group views given the above 
data. The case when the group view starts as the initial member, grows to the maximum 
number of members and then shrinks to the original host is the only possible case handled 
by the filter. Figure 22 illustrates the required group view changes. 
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Figure 22 Group Changes Required by the Filter Program 


A further restriction is that all members departing the group must be the host's 
acwnbr. This removes the need for a local database in the filter program to track which 
members are part of the current iteratior of timestamp evaluations. Future improvements 
to the filter program can implement a dynamic database to account for all possible changes 


to the test group. The format of the output file is shown in Figure 23. 





Figure 23 Time Stamp File Format 


The time required to implement a change at a given member i is calculated by 
subtracting the initial time stamp from the completion time stamp. 
ti = tes —lar Lit host 
ti = bes —tai | i = host 


The average time to commit a change at each member was calculated as follows: 
A 


However, as the group increases from m members to n+] members, the joinagree 


must be processed by the n members currently in the group. Similarly, the n members 


32 


must receive and process the joincomit token. Hence, for a join to a group of n members, 
the processing time must be n(tagree + tcomit), where f,,,,, is the time required to process an 
agree token and t 


comit 


is the time to process a commit token. The communication time to 


transmit the token from one member to its neighbor is t,,, Since each of the tokens must 
be transmitted to the n members, the total communication cost is t=2n* (tcomn). 


Therefore, the total time required to implement a join at all members of a n member group 


is 
t= N(Lagree + tcommit + 2tcomm) 


Notice that the time to implement a change is proportional to the size of the group. 


Table 4 PROCESSING TIME VALUES 
Occurrence 


time to process an agree token 
time to process a commit token 
inter-member communication time 


Similarly, the expected time for a failure can be determined to have a linear 


relationship to the size of the remaining group. 


B. TESTING 

The performance of the GMP was tested on the ECE LAN consisting of SUN 
workstations linked via an Ethernet. There were no gateways between any of the 
members of the group. Only single complete changes were allowed at any given time. A 
complete reconfiguration included the underlying FIFO channel as well as the logical ring 
structure. A linear relationship was observed between the number of members in the 
group and the average time it took to commit the new member. Figure 24. Since the 
communication depends heavily upon the network load as well as the individual processor 
load, average values over a large variety of conditions such as time of day and number of 


people on the network were generated in order to get reliable data points. 
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Figure 24 Average Time for each Member to Implement a Join to the Group 


Similarly, the time required to remove a member from the group view was obtained and 
plotted. Thus, the relationship between time and group size was determined for a 
decreasing group size. Again, only single complete changes were allowed. Figure 25. A 


linear relationship was observed as expected. 


Average Time (seconds) 


9 8 7 6 
Final Group Size 





Figure 25 Average Time to Implement a Failure in the Group 














V. METHODS FOR IMPROVING PERFORMANCE 


In this chapter we explore two methods that might be used to further improve the 
overall performance of the protocol. Each could be incorporated along with the other or 
individually. 


A. MESSAGE REDUCTION 
The protocol as currently implemented has a high overhead due to the number of 
inter-member messages required to effect changes in the logical ring structure. In order to 
reduce the number of messages required for the maintenance of the logical ring, structure 
three methods are discussed below. 
1. TokenPool versus Tokens 
Consider multiple near simultaneous changes to the group view. Transmitting 
the token pool instead of individual tokens will result in a reduction of messages if the 
changes occur close enough together such that the token pool for one change includes the 
tokens for the subsequent changes. This will result in a decreased number of messages. 
However, the probability of such changes occurring is minimal. Since changes to the 
group are uncorrelated, the probability of such changes occurring is minimal. The 
corresponding reduction of message traffic is negligible. Additionally, there is an increase 
in the size of the message for most traffic. Modifications required to effect this include the 
dynamic generation of the group view by the FIFO channel. Instead of receiving the token 
pool for generation, the FIFO channel would receive a flag and, at that point, generate the 
external token pool message. One such method might be to set a flag that indicates that 
the token pool must be transmitted. FIFO properties are maintained by the order in which 
the tokens are processed upon receipt. Reduction will occur only if multiple changes 


occur prior to the transmission of the token pool for the initial change. Problems arise in 


the correct setting of the transmit flag; i.e., did change #2 get sent in the last token pool, 








or is another token pool transmit required. This method is not recommended since the 
number of messages is reduced only in special cases. 
2. Periodic Token Pool 

Recall that tokens are generated only if a change to the group view occurs. 
Another method that will reduce the number of inter-member messages is to periodically 
send the token pool instead of individual tokens. In this manner, multiple tokens can be 
transmitted simultaneously. Message reduction is indicated only if multiple changes to the 
group view occur within the period of the token pool transmission. If this does not occur, 
message traffic will actually increase; i.e., if there are no changes within this period the 
token pool is still transmitted. This method will also increase the latency in phase 
completion as tokens are not immediately forwarded around the ring. Additionally, the 
message size will be increased. This method is not recommended either. 

3. Piggyback the Token Pool 

A further refinement would be to include the local token pool as part of the 
status report. The monitoring member, upon receipt of a statusrpt, would parse the token 
pool and process the appropriate tokens. FIFO channel requirements are maintained by 
the order in which tokens are processed upon receipt of a token pool. This, however, 
leads to additional processing for every status report. 

Difficulties might occur in the latency of cycle completion in a large group. 
Most notably, consider when a new member has requested to join an existing group. 
Define ¢, as the latency within a process. It is the difference in time between receiving the 
token pool via a statusrpt and transmitting the tokens around the ring. 1, can be modeled 
as a random variable. Ignoring the communication time, ¢..,, and the processing time, 
torocr aSSOciated with the normal processing of tokens, the latency for a change to an N 
member group (i.e. a join request) becomes: 

T oat = 2(N X t1) 

Thus, the latency involved may become prohibitive for large N. This method is 


recommended for implementation, provided that the latency of changes is not important. 
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Care must be taken to ensure the time-out on a join request is large enough to encompass 
the worst case scenario. However, since the time-out is finite, this method place an 
implicit upper bound on the maximum group size. Once the group is large enough, new 
members will be unable to join due to the time needed to implement the join around the 


ring. 


B. SINGLE-THREADED PROGRAM 
1. Problem 
The current design of the protocol involves concurrent processes handling 
specific areas of responsibility. Fully implementing this design would allow each process 
to reside on different processors. However, in most cases, a single processor is the norm. 
There is a significant amount of overhead due to the context switching and intra-member 
messages. The asynchronous nature of the protocol has several areas requiring process 
blocking as mentioned in chapter II. 
2. Solution 
Redesign the main process using a single-threaded program . Figure 26 shows 


the recommended processes and inter-dependencies. 


TOKEN 


PROCESSOR 





Figure 26 Single-Threaded Process Inter-Dependencies 


3. Justification 
The TokenProcessor would be a single-threaded program combining all aspects 


of the current design as shown in Table 4. The watchdog timer must still be a separate 
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entity by definition. The FIFO channel is not included in the TokenProcessor as Back and 


Front, are by nature, separate programs without any timing restrictions. 


Table 4 SINGLE THREADED PROCESSES AND EQUIVALENTS 


Timer Timer 


JoinProcessor 
AgreeProcessor 
ComitProcessor 
IntegrateMember 
TokenProcessor 
Status Table Manager 
Group View Manager 
Token Pool Manager 


StatusMonitor 
StatusReporter 


BACK 
FRONT 





The asynchronous nature of the design would be eliminated. The need for block 
and wait would be eliminated if a single-threaded program design were to be used. 
Additionally, the design would result in a significant decrease in the overhead costs due to 
the inter-process messages and connecting services being eliminated. The single-threaded 
program also eliminates the need for separate database managers. The TokenProcessor 
can maintain all databases internally, with different pointers keeping the different databases 
separate. 

Concurrent with the new design, a review of all subroutines is recommended. 
Current design and programming practices preserves all data passed to the subroutines. 
This can lead to significant overhead since the data is stored multiple times. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 


In this thesis, the modifications required to implement the group membership 
protocol as proposed by [5] are presented. The protocol has been successfully 
implemented and to date has run continuously for more than 48 hours with a stable group 
membership. Additionally, a group size of 20 members was achieved. These results, 
though preliminary, are the first for this protocol. Although the protocol is functioning, 
continued debugging and improvement are currently going on. 

As expected, the time required to implement a change was found to have a linear 
relationship to the eventual group size. 

Further work should include the re-design of the protocol as a single-threaded 
program. In this manner, the response of the protocol can be improved as inter-process 
communication time is reduced drastically. However, attempting to implement the 
message reduction schemes to improve performance is not recommended as there is little 
to gain in the number of messages. On the contrary, implementation of the message 
reduction schemes would result in a large increase in the latency of changes to the 
membership. 

Additional research is suggested in the area of network partitioning. Consider that a 
network may partition in two separate halves that are fully connected on either side of the 
boundary. A group originally existing on both sides will become two groups with the 
same name operating independently on either side of the partition. The difficulty arises 
when the network is repaired. The protocol does not provide for the possibility of 
merging the two groups back into the original group. The problem of handling network 


partitioning is non-trivial. 
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Initial_ Parameters | INTEGRATE 


MEMBER | touen 
Token_Pool 


Status_Q MONITOR Status_Repo 
P PROCESS 
Join_Request 
AGREE 
PROCESSOR 


Initial Parameters 


1 Token_Acknowledge 
Status_Query 













ACWNBR NEW MEMBER CWNBR HOST 
(FRONT) (FRONT) : (BACK) (BACK) (BACK) 





Figure Al FIFO Channel - Process Dependencies 





FIFO Channel - FRONT Process 





1 Wait for a channel ready to read 
2 if (external channel ready) 
3 if (Status_Query) 





4 send Status Query to MonitorProcess 

5 _ else if (JoinRequest) 

6 send JoinRequest to JoinProcessor 

7 else if (InitialParameters) 

8 send /nitialParameters to JoinProcessor 
9 else if (TokenAck) 

10 if (Received_Serial_ Number = Expected_serial_number) 
11 remove Head_of_Queue 

12 decrement Queue_Counter 

13 end 

14 end 


15 else /* internal channel ready */ 
16 if (Token) 


17 change Token to external format /* add external header */ 
18 insert Token in queue 

19 increment Serial_Number 

20 increment Queue_Counter 

21 else if (TokenPool) 

22 discard all messages in queue 

23 change TokenPool to external format /* add external header */ 
24 insert TokenPool in queue 

25 increment Serial_ Number 

26 increment Queue_Counter 

27 else if (StatusReport) 

2: update cwnbr 

29 send StatusReport to cwnbr 

30 end 

31 end 


32 if (Queue_Counter > 0) 

33 send Head_of_Queue to cwnbr 

34 set Expected_serial_number = Head_of_Queue_serial_number 
35 end 





Figure A2 FIFO Channel - Front Process 
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FIFO Channel - BACK process 
























1 Wait for a channel ready to ready 
2 if (internal channel ready) 
if (Status_Query) 
update acwnbr 
send Status Query 
else if (/nitial_Parameters) 
update acwnbr 
send Initial Parameters 
else if Join_Request) 
10 send Join_Request 
il end 
12 else /* external channel ready */ 
13 if (message originator = acwnbr) 
14 if (Status Report) 
15 send Status Report to MONITOR_PROCESS 
16 else if (Token) 
17 if (Serial_Number = Expected_Serial_ Number - 1) 
18 send Token_Ack /* to acwnbr */ 
19 end 
20 if (Serial_Number = Expected_Serial_ Number ) 
21 send Token to AgreeProcessor 
22 send Token_Ack /* to acwnbr */ 
23 increment Expected_Serial_ Number 
24 end /* out of order messages are discarded */ 
25 else if (Token_Pool) /* Token_Pool is always accepted */ 
26 send Token_Pool to AgreeProcessor 
27 send Token_Ack /* to acwnbr */ 
set Expected_Serial_ Number = Serial_ Number + 1 


OO A MN a W 

















Figure A3 FIFO Channel - Back Process 
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Agreement Integrate 
Processor Member 


Block 
Status_Table_Request Unblock 


Token Pool 
Manager 
Token_Pool_ 
Request 


Figure A4 Monitor Process - Internal Structure and Dependencies 


ReportStatus process at p, 


1 if (not blocked by IntegrateMember) 

2 if (querying member € GV,, or has joinagree status) 
3 Pron = Querying member 

4 send status to p,_,, 

5 if (previous querying member = p,_,,) 

6 send TokenPool(p,) to p,o.n 

7 end 
8 end 
9 end 


end ReportStatus 


Figure AS Reporting of Status 
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Figure A6 Agreement Processor - Process Dependencies 
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AgreeProcessor for agree,{p,) at p, 
if (not blocked by CommitProcessor) 
if (initiate agreement message received) /* p, = p; */ 
add agree,(p,) to TokenPool(p,) 
ST,(p,) <— joinagreed or failagreed 
send agree,(p,) to cwnbr(p,) 
send acknowledgment to calling process 
else /* a token or external token pool is received */ 
if (ExtTokenPool) 
for Vtokens € ExtTokenPool 


if (token € TokenPool(p,)) 
if (originator failed) 
ProcessToken 


end 
else /* token not in TokenPool */ 
if (received for the first time) 
ProcessToken 
end 
end 
end 
else /* a token was received */ 
if (received for the first time) 
ProcessToken 
end 
end 





Figure A7 Agreement Processor 
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LostAgreeToken 
if (joinagree) 
if (rank(p,) > rank(p,)) 
return true 
else 
return false 
end 
end 


if (failagree) 


if (RelativeRank(p, , p,) > RelativeRank(p, »P)) 
return true 

else 
return false 





Figure A8 Determination of Token Originator's Failure 
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ProcessToken 
if (joinregst) 
send token to JoinProcessor 
elseif (commit) 
send token to ComitProcessor 
elseif (agree) 
if ((p,; # P) && (agree token ¢ TokenPool(p,)) 
add agree,{p,) to TokenPool(p,) 
ST,(p,) <— FailAgreed or JoinAgreed 


else P; 
if ((p, = p) | (Vp, | p,>p, Pp, € ST) 
compute rank Vp,é ST,; with Agreed status 
















send agree,{p,) to cwnbr(p,) 


if rank(p,) = smallest 


send initiate_comit to ComitProcessor 


else 


Figure A9 Processing Agree Tokens 
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STp{p,) <— joinpendg or failpendg 
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Figure A10 Commit Processor - Process Dependencies 
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CommitChange for commit, (p,) at p; 

/* Depending on whether a join or departure */ 

1 add or delete p, from GV(p,) 

2 delete p, entry from ST, 

3 vnlp.) — va(p) + 1 

4 delete all commit tokens received before agree,(p,) from TokenPool(p,) 
5 if Goin committed && joinreq,(p,) € TokenPool(p)) ) 
6 

7 

8 


delete joinreq,{P,) 

end 

add commit,(p,) to TokenPool(p;) 
9 delete agree,(p,) 
10 if (current host = p,) 
11 determine new p,,, 
12 end 
13 if ((join committed) && (p,,,, = P,)) 
14 send ST,,, TokenPool(p,), and GV(p,) to acwnbr(p,) 
15 end 
16 send commit,(p,) token to cwnbr(p,) 


end CommitChange 





Figure All Actions for Committing a Change 





ProcessCommitT kn for commit,{p,) at p, 


1 if (initiate commit message received) 

2 generate commit token 

3 token to be processed <~ generated token 

4 else if ((p,#p,) && (not duplicate) 

5 token to be processed <~ received token 

6 else 

7 exit 

8 end 

9 CommitChange 

10 while ( p,€ ST), with pending status & Rank(p,) < Rank(p,), p,, € ST») 
11 generate commit token 

12 token to be processed <~ generated token 

13. CommitChange in rank order 

14 end 


Figure Al2 Generate / Receive and Process a Commit Token 
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Figure Al3 Integrate Member - Process Dependencies 


IntegrateMember 

if (initial parameters) 
send blocking message to status reporter 
send GV to group view manager 
send unblocking message to status reporter 
send ST to status table manager 
send TokenPool to token pool manager 


else 


get GV,, from group view manager 
get ST,, from status table manger 
get TokenPool(p,) from token pool manager 
assemble initparam message 
send initparam message to new member 
13 end 








Figure Al4 Integrate Member Process Specification 
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InitiateJoin for a join request message/token for p,_, at p, 


while (true) 
if (p_, € STp, GV>,) 
receive join request message or token for p_ 
end 
if (P; ra Prost) 
send initiate agreement message to AgreeProcessor for p_, 
block until AgreeProcessor acknowledges end of processing 
else 
ST p(Prw) €— JoinRequested 
if (join request message) /* p_, locates p, and sends its join request */ 
generate joinreq,(P,_,) token 
end 


add joinreq,(p,,,) to TokenPool(p,) 
send joinreg token to cwnbr(p,) 


end InitiateJoin 


Figure Al5 Processing of a Join Request Message / Token 
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Figure Al7 Status Table Manager - Process Dependencies 
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Figure Al8 Token Pool Manager - Process Dependencies 
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